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The Robert Wood Johnson Foundation requested this utilization-focused evaluation of its 
Active Living Research (ALR) program. This evaluation reports on the trajectory of 
influence of past and future ALR outcomes on field-building and policy contributions as 
well as on possible users of completed and disseminated ALR products. 


In 2006 and 2007, key-informant interviews were conducted with 136 representatives of 
first-line potential users of ALR research products, including state physical activity and 
nutrition program coordinators, policymakers, scientists, and funders. Literature reviews, 
bibliometric analyses, and document reviews served to describe the context for ALR’s work and 
the ways it could enhance its utility for field building and policymaking. 


The contributions of ALR to the emerging transdisciplinary field included leadership in the 
development of measurement tools, epidemiologic studies, implementation research, the 
translation of research to practice, and the communication of learned lessons to diverse 
audiences. ALR’s contributions to policy discussions were found across a spectrum of 
policy-development phases that included describing the problem, raising awareness of alter- 
native strategies for increasing physical activity, convening nontraditional partners, and 
evaluating policy implementation. 


Policy-relevant research can make contributions to policymakers’ thinking but almost never 
causes a change by itself. Five years after the original authorization of ALR, there is ample 
evidence of its recognition as a resource by key players, its field-building influence, and its 
contributions to policy discussions. All these bear promise for a broader contribution to obesity 
prevention. Recommendations for increasing ALR’s impact on policy and practice are offered. 
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Introduction 


ctive Living Research (ALR) is one of five grant- 
making programs launched by the Robert Wood 
Johnson Foundation (RWJF) in 2001 to increase 
physical activity at the population level.’ This suite of 
programs sought to increase physical activity through 
research, community demonstrations, technical assis- 
tance, and policy and environmental changes to create 
more activity-friendly environments. From its inception, 
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ALR was designed to “stimulate and support research to 
identify environmental factors and policies that could 
substantially increase levels of physical activity among 
Americans of all ages, incomes, and ethnic backgrounds” 
(www.activelivingresearch.org). While ALR’s emphasis re- 
mains focused on the built environment, it was redirected 
by the RWJF in 2003 to identify policy and environmental 
determinants of physical activity within the low-income 
and racial/ethnic populations where levels of childhood 
obesity are highest and rising fast. For more information 
on ALR, the reader is referred to the articles in this 
supplement to the American Journal of Preventive Medicine 
by Orleans et al.! and Sallis et al.,? the latter of which 
includes findings from internal evaluations conducted by 
the ALR National Program Office (NPO). 

By 2006, ALR was completing its first funding cycle and 
was being considered for renewal, at approximately the 
same time that the Healthy Eating Research (HER) 
program joined the suite of RWJF programs focused on 
the prevention of childhood obesity. It was at this juncture 
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that the RWJF called for the external evaluation of ALR; 
this paper reports on one of two such evaluations. 


The Evaluation’s Charge and Guiding Questions 


To assess ALR’s current and future contributions and to 
inform decision making by the RWJF and the ALR NPO, 
two complementary external evaluations were commis- 
sioned by the RWJF. One evaluation, reported also in this 
supplement,’ assessed ALR process and outcomes. The 
other evaluation, reported here, was a utilization-focused 
evaluation to assess and comment on the trajectory of 
influence of past and future ALR research findings as well 
as on the possible users of completed and disseminated 
ALR products. Utilization-focused evaluation is a “ 
process for helping primary intended users select the 
most appropriate content, model, methods, theory and 
uses for their particular situation.”* In keeping with these 
principles, the internal stakeholders of the RWJF chose 
two evaluation questions that would be useful to their 
deliberations: 


1. To what extent, and how well, is ALR working to build 
the field of research and practice focused on policy 
and environmental factors conducive to physical 
activity? 

2. To what extent, and how well, are research findings 
from ALR contributing to policy discussions about 
how to promote physical activity through policy and 
environmental changes? 


A Focus on Knowledge Utilization 


In asking these questions, the RWJF chose knowledge use as 
a legitimate intermediate objective for ALR after 5 years of 
operation. While other evaluations demonstrated ALR’s 
production of research knowledge,” this evaluation was 
intended to assess the potential use of that research 
knowledge: How do—and will—ALR research products 
contribute to a field of research and practice, including 
policy? This focus on practice was added to the original 
RWJF question to enable an exploration of field building. 
Further, in the tradition of policy-related research where 
the “results are never self-explanatory,”” the RWJF asked 
the evaluation team to provide commentary about a 
trajectory of use. 

A focus on knowledge use is consistent with evaluation 
and policy research over the past 3 decades*®” that has 
discredited the idea that research alone has a direct and 
immediate effect on policy decisions.* Such instrumental 
use® was found unrealistic, and probably undesirable, 
given the failing of much research to influence practice, 
much less policy. Policymakers take research information 
into consideration along with a large number of other 
types or sources of information and issues, such as the 
media, constituents, and colleagues.’ Policymakers pro- 
cess all this information with any number of documented 
models of use, including problem solving, political, inter- 
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active, tactical, or enlightenment models.!° It is an en- 
lightenment model that the RWJF brings into play in this 
evaluation where it seeks an understanding of the options 
and assumptions underlying lines of action as they apply 
to decision makers within the RWJF and to the external 
recipients of ALR research products. 


A Tale of Trajectory and Contribution 


Following an explanation of the methodology used, eval- 
uation findings and commentary are presented in a series 
of questions that inform the guiding questions asked by 
the RWJF. The broad context of field building is explored 
and then narrowed to a focus on ALR’s potential contri- 
bution within that context. Consistent with the knowl- 
edge-utilization literature, the evaluation did not look for 
a direct or sole influence of ALR findings on field 
building and policy change, but rather looked at ways that 
ALR research could enlighten the policymakers and prac- 
titioners who link research with policy discussions. Indica- 
tors of enlightenment included whether such practitio- 
ners and policymakers were aware of ALR products and 
saw the relevance of those products to their responsibili- 
ties. Findings of policy contribution are organized around a 
conceptual framework that defines contributions broadly 
and at multiple stages in the policy process. Lastly, a 
discussion of the findings and recommendations are 
offered. 

The evaluation took into consideration the relative 
youth of ALR as well as the RWJF’s recent expansion into 
childhood-obesity prevention, encompassing policy and 
environmental approaches to both active living and 
healthy eating. As described by Hirschhorn and Gil- 
more,'! philanthropy builds a field of research, practice, 
or social change from ideas that are in currency at the 
time. It takes time for research to be proposed, funded, 
produced, and disseminated to policy and practice audi- 
ences. This study, therefore, assessed the recognition and 
the perceived contributions and utility of the ALR-funded 
research for field building and policy. It did not expect to 
establish causation or attribution; it explored implemen- 
tation and policy contexts, and it offered recommenda- 
tions on potential ways the initiative could be strength- 
ened to influence policy and practice. 


Methods 
Logic Model 


Using recommended evaluation practice,'* evaluators devel- 
oped a conceptual logic model to map an understanding of the 
interplay of various social and institutional forces that bear on 
policy contribution and field building. Construction of the logic 
model (Figure 1) was based on ALR’s own logic model*; on 
multiple understandings of research influence and knowledge 
utilization’® such as diffusion,’* implementation,” and transla- 


tion'®; and on the stages and typologies of the policy process.'” 
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Figure 1. Evaluation logic model to assess ALR’s contributions to field building and policy 


The left side of the logic model identifies five domains of 
contribution to field building: policy, society, scientific commu- 
nity, professional community, and funders. Across time, the 
model anticipates individual sector trends that will be reframed 
and infused with policy in ways that support the emergence of a 
new field. The shaded path embeds ALR in multiple scientific 
contributions to field building. Space does not permit a full 
discussion of this model, but its presentation here is critical to a 
transparent explanation of the sample, methods, and analyses 
chosen. 


Sample 


The study collected data largely through telephone interviews 
with key informants. The sample consisted of 136 respon- 
dents. An outside-inward approach to sample selection is 
represented in the logic model, Figure 1, with the RWJF’s 
intended impact and ALR’s intended users. These end-users 
included (1) the most important first-line consumers and 
implementers of ALR research products at state and local 
levels; and (2) policy shapers, including advocacy, federal, 
and philanthropic organization leaders who ideally would 
apply those products in broad national policymaking. Inter- 
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views were conducted with informants representing four of 
the five levels of the logic model, as follows. 


Professional community. This comprised 50 state coordina- 
tors employed by the 28 states funded by the CDC Nutri- 
tion and Physical Activity Program to Prevent Obesity and 
Other Chronic Diseases,‘* plus nine coordinators em- 
ployed by states not funded by the CDC. At the nexus of 
federal, state, and local efforts in physical activity and 
obesity prevention, these coordinators represent the most 
likely translators and disseminators of research to frontline 
users of ALR knowledge and products in state and local 
policymaking. Further, they represent a manageable, na- 
tional population of one kind of professional community; 
the 50 coordinators were a sample of 84 CDC-funded 
positions (some unfilled) at the time of the study. 


Policy. State coordinators identified 36 policy shapers and 
advocates via a snowball sample as key players in the field and 
another type of potential end-user of ALR products. They 
included administrators or other key representatives of agen- 
cies engaged in children’s health and advocacy, health policy 
development and analysis, land-use planning, media and 
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aggregated by key-informant type, 
and frequency distributions were 
calculated. These data were used 
to examine assumptions about 
the presumed linkages shown in 
the logic model (Figure 1) and to 
identify patterns across key-infor- 
mant levels. To answer the ques- 
tion of policy contribution, the 
analysis sought to trace practice 
influences on intended users 
inward toward ALR or other 
sources of influence (moving 
from right to left in the logic 
| IV model). To answer the ques- 
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Figure 2. Science—policy—public— evaluation spectra: a framework to assess policy contributions 


communications, parks and recreation, physical activity advo- 
cacy, and transportation planning. 


Scientific community. Twenty-three researchers were identi- 
fied through literature reviews as key contributors to the 
fields of nutrition, physical activity, obesity control, and 
health promotion. These were individuals who conducted 
population-level policy and environmental research address- 
ing one or both sides of the energy-balance equation (i.e., 
physical activity and calorie intake) and had knowledge of 
policy development and field building. 


Funders. These were seven representatives from philan- 
thropic, health-related foundations other than the RWJF who 
have a role in supporting research and/or demonstrations in 
the areas of physical activity, obesity, or children’s health. In 
addition, interviews were conducted with 11 RWJF leaders, 
managers, and key consultants. 


Data Collection 


Telephone interviews with key informants were conducted 
between August 2006 and April 2007. Each lasted from 15 
minutes to 60 minutes. A standardized protocol reflecting 
the evaluation questions and major constructs of the logic 
model was developed with variations for each type of key 
informant. The protocols contained both closed- and open- 
ended questions. Core questions asked of more than one 
group of informants inquired about awareness of ALR, 
examples and indicators of ALR’s policy-relevant contribu- 
tions and field-building efforts, and larger contextual influ- 
ences on ALR methods and impacts. In addition, profession- 
als and policy shapers were asked questions about the general 
influence of research on practice and policy and about ALR 
specifically; researchers were asked questions about field- 
building facilitators, barriers, and trajectories; and funders 
also were asked about field building and about the sustain- 
ability and direction of the field. 
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tion of field building, the anal- 
ysis moved from ALR outward 
to other sectors. To analyze 
contribution to policy, an addi- 
tional framework, the Science- 
Policy—Public—Evaluation Spectra 
(Figure 2), was developed and will be explained in more 
detail later. 

Qualitative data from interviews were analyzed for patterns 
within and across constructs using content analysis.'? Conclu- 
sions or inferences for recommendations were drawn from two 
or more consistent and corroborated interviews and observa- 
tions. Most of these could also be corroborated by quantitative 
analyses of the larger samples of interviewees. 

Combined quantitative and qualitative data were analyzed in 
stages: (1) preliminary findings were reviewed and interpreted 
by the evaluation team; (2) data were triangulated”® or verified 
across multiple sources; and (3) preliminary data interpretation, 
findings, and conclusions were verified or corroborated in 
discussions with RWJF leadership and staff, the ALR NPO, the 
ALR NPO advisory committee, and ALR grantees. These reviews 
of findings and interpretation are consistent with member 
checking”! and with a participatory approach to evaluation 
research. 


Evaluation 


Literature Review and Bibliometric Study 


A professional librarian conducted online searches of the public 
health, law, environment, and transportation literature pub- 
lished in the previous 6 years (2001-2006) on the keywords 
obesity and overweight, physical activity and the built environment, and 
public health policy development. She also conducted a bibliometric 
analysis covering the period 1975-2007 to identify patterns in 
the publication of scholarly articles for evidence of the emer- 
gence of multidisciplinary fields related to physical activity and 
nutrition. (Bibliometrics is the quantitative study of publications 
to profile scientific disciplines and includes the analysis of 
citation counts over time.”***) This analysis involved searches 
of the National Library of Medicine’s PubMed database 
(using the medical subject headings [MeSH] thesaurus), and 
the Institute for Scientific Information’s Web of Knowl- 
edge™ database. The search also examined evidence of the 
merging of active living research with seemingly disparate 
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concerns such as climate change and the conservation of 
carbon-based energy consumption. 

Using PubMed, the study examined patterns in the publica- 
tion of scholarly articles for evidence of the emergence of 
transdisciplinary fields of research comparable to the fields of 
health promotion and tobacco control. In particular, the search 
looked for the trajectory of interest in childhood obesity and the 
sources of financial support for studies on nutrition and physical 
activity. The study used the Web of Knowledge database to 
determine where articles on physical activity and the environ- 
ment (defined broadly) are being published and which disci- 
plines the authors of each article represent. Finally, the search 
noted the publication venues of ALR studies and tracked 
whether websites related to active living linked to the ALR NPO. 

These latter analyses served more to provide a broad under- 
standing of the national and temporal context and trajectory of 
active living research than to evaluate the first few years of ALR 
research use. Both the qualitative interview data and the biblio- 
metric analysis substantiated or modified the evaluation’s logic 
model and provided insights on perceived needs and on oppor- 
tunities in the field for research to influence policy discussions. 


Results 
Is a Field Emerging? 


The evaluators defined field building as the develop- 
ment of a critical mass of effort, energy, and capacity to 
offset the forces that promote physical inactivity, poor 
nutrition, and excess weight. This is consistent with the 
characterization of field building by the RWJF to build 
leadership and capacity by strengthening infrastruc- 
tures, supporting research, and encouraging linkages 
among researchers, practitioners, and policymakers— 
and, in its Human Capital portfolio, which referred in 
2007 to “engaging researchers from a wide range of 
disciplines” and “building expertise, to address the 
social, environmental, behavioral, economic, and 
biological factors that affect health.”** 

The evaluation found ample evidence of emerging 
elements of what usually comes to be identified as a 
field—in this case, of research and action on some 
combination of physical activity, the built environment, 
healthy eating, and childhood-obesity prevention. The 
evidence included a growing body of scientific literature, 
growing acknowledgement of the importance of the obe- 
sity epidemic and related problems that this literature is 
addressing, and growing consensus on the need for 
research to guide both policy and the shifting research 
and action agendas of funding agencies. 


A growing literature. The initial study objective, without 
attempting to attribute the growing literature and other 
scientific and policy movement to ALR, was to understand 
the context in which ALR was launched and active during 
its first 6 years and to ascertain a possible trajectory of field 
building that the RWJF could help nurture in future years. 
Between 2000 and 2006, a period during which relevant 
RWJF programs were being carried out, the annual num- 
ber of journal articles on childhood obesity showed a 
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fourfold increase. Sallis et al.? confirm these findings, as 
shown in their Figure 2. Also found was the noteworthy 
trend of the increasing publication of physical activity 
articles in journals for fields other than health, including 
transportation, environmental science, and urban plan- 
ning. While only 16% of the studies funded through the 
ALR program had been completed at the time of this 
evaluation, nearly half of the researchers funded in the 
first six rounds of ALR were actively publishing the results 
of studies about physical activity and environments. 


Research support. The bibliometric analyses of the field- 
building question also addressed the kind of balance that 
has been struck between physical activity and nutrition 
research in relation to obesity control and identified 
sources of support for that research. A PubMed search of 
the periods 1976-2000 (n=270 citations) and 2001-2006 
(n=321 citations) found that the percentage of support 
for research that was focused on physical activity alone, 
and on physical activity and nutrition combined, in- 
creased relative to support for research that was focused 
solely on nutrition, reflecting the growing attention to 
physical activity. Also, during the same periods, the per- 
centage of nonfederal support of research in these areas 
increased from 18% to 27%. It appears that philanthropy 
has played an increasing role in supporting this research. 
In a review of the ALR grantee abstracts, it was noted that 
several acknowledged that their funding from RWJF was 
(or would be) supplementing prior funding from a fed- 
eral government source. Since this review, the RWJF has 
entered into an important three-way collaborative rela- 
tionship with the NIH and the CDC to facilitate greater 
involvement of prominent federal funding sources.”” 


What Is the Name of the Field? 


Key informants could not agree on the name of the 
emerging field that included ALR, HER, and other obe- 
sity-prevention approaches. They suggested names such as 
energy balance, active living/healthy eating, and promotion of 
healthy living. One funding key informant captured both 
the frustration of trying to name a field and the challenge 
of deciding which disciplines would come together to 
create it: 


It’s not one specific field, it is many fields. We 
don’t need to find an umbrella that everyone 
stands under all the time. There is a need to 
facilitate linkages around specific issues. The work 
is building fields, not a field. 


The bibliometric analysis also found increasing in- 
stances of titles and abstracts combining topics related to 
active living with topics related to climate change and 
carbon-based energy consumption. Whatever the field is 
named, it will evolve with the changing epidemiology and 
economics of the problems it addresses along with the 
academic, professional, and political debates surrounding 
them. 
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Is There Awareness of ALR Within the Field? 


Key informants recognized that RWJF programs (with 
special emphasis on ALR) had played a role in developing 
the science related to active living. The RWJF received 
recognition for leadership in the development of mea- 
surement tools, implementation research, the translation 
of research results to practice, and the communication of 
lessons learned to diverse audiences. Table 1 summarizes 
the recognition and perceived influence of ALR by key 
respondents. A high percentage of all three types of 
intended users had heard of ALR. Among state coordina- 
tors who had heard of ALR, nearly two thirds were aware 
of at least one ALR research study. Nearly half of the 
coordinators who were aware of an ALR study and more 
than two thirds of the policymakers and scientists who had 
heard of ALR had had contact with an ALR researcher. 
This contact stemmed from the involvement of research- 
ers in community settings such as meetings, committees, 
and task forces. At least for state coordinators, the CDC 
played an active role in fostering such research— 
practitioner exchanges. All three types of informants had 
visited the ALR website. All scientific key informants who 
had heard of ALR had read an ALR publication; two 
coordinators subscribed to the ALR newsletter. 


What is the trajectory of the field? The study team was 


asked to assess and comment on the trajectory of the 
active living field in relation to their experience with 
health promotion and related fields. Without a control 
group or other contemporary comparison for the evalu- 
ation of ALR’s contribution to field building, the study 


Table 1. Percentage of positive responses to questions 
about awareness of ALR and use of selected ALR products 
by type of respondent 


State 
coordinators Policymakers Scientists 
n(%) n(%) n(%) 
n=59 n=36 n= 14* 
Heard of ALR 53 (90) 28 (78) 10 (71) 
Aware of at least 38 (64) na na 
one ALR study 
n=37? n=28° n= 10° 
Had contact with 17 (46) 18 (64) 7 (70) 
ALR researchers 
n=55 n=28° n= 10° 
Visited ALR website 40 (73) 11 (39) 7 (70) 
n=54 
Subscribe to ALR 2 (4) na na 
newsletter 
n= 10° 
Read ALR na na 10 (100) 
publications 


“Asked of a subsample of scientific key informants whose awareness of 
ALR was unknown 

Asked only of key informants who were aware of at least one ALR 
study 

“Asked only of key informants who had heard of ALR 

ALR, Active Living Research; na, question not asked 
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turned to tobacco control as a recent example of both 
field building and public health success in reversing a 
national epidemic of like proportions with some similar 
causal forces.”°?” The trajectory of the field encompassing 
active living, childhood obesity, and the environmen- 
tal and policy influences on these—as inferred from 
the bibliometric analysis and a tracking of state legislative 
initiatives—resembles the acceleration phase of the S- 
shaped diffusion curve of tobacco control, clean-air initi- 
atives, and restrictions on cigarette-vending machines.”°** 
The growing attention to a balancing of the behavioral 
aspects of health, including nutrition and physical 
activity, with environmental determinants of those be- 
haviors could be seen as the continuation of growth in 
the general health promotion field. The fields of active 
living and obesity control, as distinct from their earlier 
incarnations in physical education, sports medicine, 
nutrition counseling, and other individual approaches, 
began their trajectories later than tobacco but currently 
appear capable of accelerating at a projected pace 
equal to or greater than tobacco control. 


Within This Field, Has ALR Contributed to 
Policy Discussions? 


To assess progress toward policy contributions by ALR in 
concert with the suite of related communications and 
implementation programs funded by RWJF, state coordi- 
nators and scientist key informants were asked whether 
they were aware of any instances in which research funded 
under the ALR program has influenced policy discus- 
sions. Of the 81 respondents to this question, one quarter 
indicated that they were aware of an ALR contribution to 
policy, nearly two thirds were not aware of a policy 
contribution, and the remaining respondents were un- 
sure. These responses are heavily dependent on the key 
informants’ understanding of contribution to policy. For 
example, it was unclear whether this phrase meant the 
passing of legislation, the raising of community awareness, 
or something else to respondents 

To capture a broad and fair understanding of con- 
tribution to policy, the study logic model was supple- 
mented with an additional framework specifically fo- 
cused on policy contribution. The Science—Policy— 
Public—Evaluation Spectra (Figure 2) is grounded in 
policy literature, in an understanding of multiple paths 
to outcomes,!*?9*° and in the data from this study. 

Expanding on Downs’s spectrum of the public’s atten- 
tion to issues,*! the framework added comparable spectra 
for policy, science, and evaluation. The four-stage policy 
spectrum includes pre-policy, developing policy, policy 
enactment, and post-policy enactment. The comparable 
spectrum for science ranges from basic to applied re- 
search. The evaluation spectrum ranges from process 
assessment to outcome-evaluation research. The frame- 
work posits that policy is a process, not an event, and that 
contributions to policy can be made at multiple points 


Am J Prev Med 2009;36(2S) $39 


across time in the context of public and scientific influ- 
ences and useful evaluation findings. 

With this analytic framework we sought to identify 
whether and how ALR had contributed or could con- 
tribute at the four stages in the policy process, with the 
recognition that ALR program staff and grantees were 
and are prohibited from using RWJF funds for lobbying 
for policy change and legislation; they are limited to 
informing the policy debate or discussion. 


Pre-Policy: Problem Described, Causes and 
Consequences Studied, and Trends Traced 


While the public may have been largely unaware of the 
developing obesity epidemic in the 1990s, research and 
surveillance demonstrated the rapid rise in obesity, espe- 
cially childhood and adolescent obesity. In the late 1990s, 
the data were used to flag the problem as one of epidemic 
proportion and worthy of public health attention. The 
data were used also to highlight jurisdictional compari- 
sons, which in turn made the problem more salient for 
state and local policymakers. As obesity rates continued to 
rise, researchers were studying the issues, the media 
disseminated the discoveries, and funders were support- 
ing studies to understand the scope and nature of the 
problem.*” The RWJF physical activity and related obesity 
initiatives, including ALR, grew out of this phase in the 
late 1990s. During the next stage, ALR made (and con- 
tinues to make) contributions to policy discussions—as 
noted in the Sallis? and Gutman’ articles—such as the 
following: 


e developing, refining, and validating research tools to 
study and describe the problem; 

e formulating and issuing requests for research pro- 
posals on policy and environmental issues related to 
obesity; 

e building research capacity to study the issues 
through funding choices, grantee training, and 
field-building activities. As one scientific key infor- 
mant said, “It’s hard to make a direct link. I have the 
impression that the products of the research are 
creating an expertise in this area”; 

e publishing research findings for the scientific 
community; 

e disseminating research findings for public and poli- 
cymaker understanding; and 

e convening annual conferences of researchers and 
presenting to them the views and perceived needs of 
policymakers. 


Developing Policy: Awareness, Mobilization, and 
Reframing the Issue 


Active Living Research contributed to the development 
of policy through increasing the awareness of physical 
activity, mobilizing partners, and reframing issues re- 
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lated to physical activity and obesity control. This is 
evidenced in the following quotes from key informants. 


Awareness/attention to the issue: 


e [cant name one specific thing. RWJF has raised the profile 
on this issue. (state coordinator) 

e ... to the degree it has raised awareness around the built 
environment, it has created a new awareness. (state 
coordinator) 


Mobilize/convene: 


e It has created [opportunities where] ... nontraditional 
partners have come together. (state coordinator) 

e Yes. At the community level. Bringing together coalitions. 
(scientist) 


Reframe the issue: 


e ... used information from the ALR... or... used informa- 
tion from fim Sallis. (multiple state coordinators, 
funders, policymakers offered such quotes) 

e ... used the financial information regarding the cost of 
physical inactivity at the ALR [web] site. (state 
coordinator) 

e ... [use] ALR research that is specific to [respondent’s state]. 
(state coordinator) 


Policy Enactment: Advocacy, Frame Policy, Pass 
Policy/Legislation 


Key informants identified a range of ways in which 
ALR products contributed to the enactment of policy 
through advocacy, the framing of policy, and specific 
legislation. This is exhibited in the following quotes. 


Advocacy: 


e Importantly, Robert Wood Johnson’s funding program is 
producing research that is generating debate (e.g., the 
debates about urban sprawl), which over time will result in 
change ... Robert Wood Johnson program is becoming very 
effective in that someone is funding an advocacy program 
(credible public speakers) to present a counterview. 
(scientist) 


Frame policy: 

e ... the walkable communities. I use this info a lot to get 
policymakers to think about it and our community. (state 
coordinator) 

e I’m less aware of the direct level the foundation has had. 
On an indirect level: use of models, research summaries in 
the Active Living by Design [have] been influential at the 
policy level. (state coordinator) 


Belief in use/potential use: 


e RWIF has funded a lot of this work and I’m sure it has had 
some policy outcomes. (state coordinator) 

e [can't give specific examples. But I know that they will. I’m 
sure the RW] study being conducted in [my state] will make 
an impact. (state coordinator) 
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Passage of legislation: 

Respondents offered six instances of ALR’s contribu- 
tion to the passage of a specific policy or legislation. 
These included Texas Senate Bill 19 on Elementary 
School Children’s Level of Physical Activity (www.legis. 
state.tx.us/BillLookup/Text.aspx?LegSess=77R&Bill= 
SB19)and Texas Senate Bill 42 on Middle School 
Children’s Level of Physical Activity (www.legis.state.tx. 
us/tlodocs/79R/billtext/html/SB00042F.HTM). These 
and the other examples were vetted with the ALR NPO, 
but not substantiated by them. It is unclear whether 
informants were mixing other Active Living programs, 
such as Active Living by Design, with ALR, or whether 
these are valid examples. 


Post-Policy Enactment: Implementation, Access, 
and Sustaining or Changing Policy 


While public interest as an issue may decline over time, 
the Science—Policy—Public—Evaluation Spectra used in 
this analysis do not assume that policy starts or stops 
with the passage of policy/legislation. From a classic 
implementation perspective, the passage of policy is a 
beginning, not an ending, and the potential for policy 
contribution continues.” 

This was the case with the Texas bills mentioned in the 
preceding section. When the ALR NPO staff were vetting 
these examples, they explained that the legislation had 
passed in 2001, the year ALR was first funded. It was 
unclear how ALR could be credited with the passage of 
those bills. However, ALR is currently funding evaluations 
of the implementation of the bills, and the results of those 
evaluations will continue ALR’s influence on policy. The 
importance of implementation as a contribution to policy 
was stressed by a policy key informant who said: 


One of the risks of research is that the emphasis on 
what is yet to be learned often gets in the way of 
implementation. I believe that the wisdom of prac- 
titioners and the community should be emphasized 
more than research. Research should be a tool to 
further the implementation of this wisdom. 


In summary, if contribution to policy discussion is narrowly 
defined as the passage of legislation, the evaluation ap- 
propriately found little or no evidence of ALR’s direct 
contribution. If contribution is understood as contributing 
to policy at multiple points in a complex and protracted 
process, then ALR did contribute, and has further poten- 
tial to contribute in ways consistent with the RWJF’s 
grantmaking goals and clear prohibitions against the use 
of its funds for lobbying purposes. 


Discussion 


A field of research and practice around active living with 
an emphasis on environmental and policy approaches 
does appear to be emerging with parallel contributions, if 
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not influence, attributable to ALR. The NPO has created 
products, linkages, awareness, and influence in its har- 
nessing of the multiple disciplines whose efforts are 
needed to effectively address active living and obesity. The 
growing literature and supported research clearly indicate 
an upswing in the number of publications and funded 
projects in which physical activity is examined and pur- 
sued as something more environmentally determined or 
policy-influenced than in the individual, behavioral, phys- 
iologic, and sports- and recreation-related approaches of 
the earlier literature. 

Key influential program planners and coordinators at 
the state level and policymakers or advocates at state and 
local levels seem to recognize ALR by name, and some 
can identify studies or initiatives that they believe were 
products of ALR that are useful to their work. Most of 
their specification of sources for basing the planning and 
policy initiative in evidence, however, are CDC sources 
and other federal documents such as the Guide to Commu- 
nity Preventive Services,*> some of which include evi- 
dence from ALR-funded studies or which ALR has had a 
hand in developing and disseminating. Many key infor- 
mants give personal credit to the NPO director, James 
Sallis, for his vision and energy in implementing ALR, but 
some were conflating ALR with other sources, including 
other programs in the RWJF active living suite and initia- 
tive for the control of childhood obesity. The CDC’s role 
in compiling and disseminating ALR research to state 
coordinators is of particular note. 

It might be possible to have the impression that the 
NPO’s original conceptual model for the ALR program 
was conceived as a one-way flow from research to end- 
users—practitioners and policymakers. This is the conven- 
tional way that researchers view the production and use of 
research products.** The research agenda and the ex- 
pectations of its application in evidence-based practice 
often fail to give sufficient credence or weight to the 
experience of practitioners, program planners, and poli- 
cymakers. The NPO has attempted through its annual 
conferences and its collaboration with various partners to 
compensate for the one-way impression of its conceptual 
model, but the research community itself has not been 
entirely prepared to break from its traditions of investigator- 
initiated and academically centered and controlled research. 

Awaiting data from long-term research projects may 
have led to missed opportunities to address targets of 
policy development that could be fruitful in the short 
term. The RWJF’s expanded investment in the Active 
Living Leadership program, now renamed Leadership for 
Healthy Communities, and RWJF’s recent initiative on 
Early Assessment of Environmental Interventions to Pre- 
vent Childhood Obesity, for example, are expected to 
enhance the policy and practice relevance of the research 
funded through ALR. The possibility of engaging the 
practice, program planning, and policy end-users more 
actively in framing the research questions and interpret- 
ing interim data in terms most meaningful and relevant to 
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their needs could signal some shift in future rounds of 
ALR calls for grant applications. Whether the size and 
scope of the current grants are sufficient to encompass 
social and environmental change effectively remains 
another question for continuing examination. 


Limitations 


Four limitations of the evaluation should be noted. First, 
the evaluation occurred early in the history of the ALR 
program and, therefore, in the early stages of impact on 
the long-term processes of policy change and field build- 
ing. Second, the evaluation does not completely reflect a 
shift in RWJF and Childhood Obesity team strategies that 
occurred concurrent with the evaluation. Third, some 
respondents confused ALR with other programs in the 
RWIJF suite of programs. Finally, snowball sampling has 
limitations in its representation of opinions, perspectives, 
and behavior that would be more divergent from those of 
the people making the referral. 


Recommendations 


Five recommendations are offered. These are based on 
the evaluation findings and the experiences of the evalu- 
ation team in the related areas of public health; health 
promotion (e.g., tobacco control); and community orga- 
nization and advocacy for policy change. 


Bridging Research and Policy 


An emerging transdisciplinary field relating the environ- 
ment and policy to active living has grown, with contribu- 
tions from the ALR NPO and grantees. Future efforts 
need to move more substantially toward the evaluation of 
promising policies and environmental-change initiatives, 
which accelerated in their passage and implementation 
during these early years of ALR. To fulfill its potential to 
make active living a reality, the emerging field needs 
substantial and coordinated investments in these areas, 
including clear incentives for researchers and other grant- 
ees to focus on the early applied and policy ends of the 
spectrum (including studying advocacy groups, legisla- 
tion, and other efforts to influence and support ALR 
policy). A national active living policy research and sur- 
veillance agenda” directed toward childhood obesity, 
with allowances for state and local variations,*° would 
serve as a useful compendium of this work and a guide to 
researchers mystified by the policy focus. 


Boosting the Visibility and Relevance of Policy 


The policy voice of ALR can be amplified by continuing to 
engage end-users and intermediaries such as CDC in 
delineating policy, surveillance, research, and evaluation 
questions*” as early as possible in the research process. 
This early focus on knowledge use increases data and 
research relevance and usefulness. For example, this 
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might involve focusing on cost effectiveness and other 
issues of particular interest to state legislators and other 
policymakers and making translation and dissemination”° 
strong expectations for every aspect of ALR work—in ways 
consistent with RWJF funding restrictions. 


Emphasizing Collaboration and Coordination 


Active living research is part of a growing transdisciplinary 
field.*’ Greater coordination among researchers, advo- 
cates, policymakers, city planners, and others at every level 
will bring needed coherence and amplify the field’s 
contributions to childhood obesity. This could include 
seeking incentives for greater collaboration across the 
suite of childhood-obesity programs (within RWJF); stron- 
ger linkages within the research community and between 
researchers and policymakers; connections to related but 
distinct fields (e.g., healthy eating research and policy); 
coordination across philanthropies (including nonhealth 
philanthropies) and across federal agencies; and a strong 
emphasis on coalition building and collaboration across 
academic disciplines and public-private sectors. The to- 
bacco experience with the systematic documentation of 
the early successes in California and Massachusetts points 
toward ways in which policy shapers and advocacy groups 
can inform the research agenda as well as be guided by it.** 


Strengthening Policy Measurement 
and Evaluation 


As recognized by the ALR NPO, the emerging transdisci- 
plinary field needs some standardization in measurement 
and evaluation tools so that research results can be more 
relevant and comparable over time, across communities, 
across disciplines, and across studies. This is especially 
true of the poorly defined and documented policy arena. 
A transparent understanding of contributions to policy is 
encouraged for future assessments, along with the use of 
a tool such as the Science—Policy—Public—Evaluation Spec- 
tra (Figure 2) to capture a range of policy contributions. 

The IOM, with RWJF, CDC, and NIH support, is 
convening a committee to develop a framework for 
evidence to monitor policy changes related to child- 
hood obesity. Other collaborative efforts are needed to 
examine how research questions are formulated (ide- 
ally including end-users and emphasizing policy) and 
how evaluation designs can take into account the many 
component parts that constitute complex interven- 
tions.** All of these could be incorporated into an 
ambitious, coherent research agenda for the growing 
field related to the prevention of childhood obesity that 
would guide research efforts and stimulate new lines of 
inquiry in the future. This work should include the 
collaboration with other major research funders, in- 
cluding the CDC and NIH, that was underway as this 
manuscript went to press. 
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Shaping Policy and Practice Through Training 


Training, technical assistance, and knowledge exchange 
offer many opportunities to shape the emerging transdis- 
ciplinary field. Training could include policy workshops 
for researchers and state legislators or other policymakers. 
Knowledge-exchange examples include providing collab- 
orative opportunities between research grantees and pol- 
icy shapers. Technical assistance examples include provid- 
ing guidance on ALR implications for new and potential 
partners working in other fields and advocacy guidance 
for everyone involved in ALR. 

Obesity as a public health problem will not decline in 
response to more research any more than individual 
weight control will yield to more information. Engaging 
the policymakers and other end-users in shaping the 
research agenda and participating in the research process 
will ensure a more-effective translation of research into 
action. 


The findings and conclusions in this report are those of the 
authors and do not necessarily represent the official position 
of the CDC. 

No financial disclosures were reported by the authors of 
this paper. 
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